Confidence Intervals and Hypothesis Testing.∗

If you have just two discrete hypotheses, then a hypothesis test is simply an application of Bayes’ Theorem. You check to see whether your data can reasonably be explained by that boring old null hypothesis (conventionally called H0), or not. To test H0, you need to compare it to one other model. One never tests a hypothesis in isolation; it is tested against an alternative (typically called H1). But, hypothesis testing is a biassed, asymmetric operation, not just a comparison of which of the alternatives is more probable. Hypothesis testing is appropriate when one of the alternatives is special: simpler, more elegant, predicted by an eminent theorist or critical to some application or conclusion. You need some prior reason to choose the null hypothesis, because it will survive the test even if it’s not the one most preferred by the data. In fact, in a hypothesis test, you will only reject H0 if the alternative is about 100(!) times more probable than H0, based on the data. Because of this asymmetry, hypothesis testing is not really appropriate when there is no special choice that we can use for H0. In such a case, it’s better to report a confidence interval, a list of reasonably probable hypotheses, or simply the most probable hypothesis. Here’s the recipe: